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Abstract —Cloud computing datacenters provide millions of 
virtual machines in actual cloud markets. In this context, Virtual 
Machine Placement (VMP) is one of the most challenging 
problems in cloud infrastructure management, considering the 
large number of possible optimization criteria and different 
formulations that could be studied. Considering the on-demand 
model of cloud computing, the VMP problem should be optimized 
dynamically to efficiently attend typical workload of modern 
applications. This work presents possible classification criteria 
for different formulations of the VMP problem from the Cloud 
Service Providers’ (CSPs) perspective in dynamic environments, 
based on the most relevant dynamic parameters studied so far 
in the VMP literature. Several examples for understanding the 
possible dynamic environments are presented for future imple¬ 
mentation of workload trace generation to deeply studies and 
further advance on this research area. Finally, future directions 
are also presented. 

I. Introduction 

The rapid demand growth for computational resources in 
modern business and scientihc applications represents several 
challenges for design, implementation and management of 
scalable datacenters to meet the requirements of customers 
in a competitive and efficient way ifffill . 

Considering the evolution of resource provisioning, three 
main models could be identihed; (1) traditional provisioning 
of resources with independent physical hardware, (2) modern 
provisioning of shared resources through virtualized hardware 
and (3) trending dynamic provisioning of resources through 
a cloud computing model a. The traditional provisioning 
environment has mostly evolved to a virtualized provisioning 
of resources in current datacenters, considering its advantages 
for management and resource utilization. 

Virtualization in modern datacenters introduces complex 
management decisions related to the placement of virtual 
machines (VMs) into the available physical machines (PMs). 
In this context. Virtual Machine Placement (VMP) represents 
the process of selecting which VMs should be executed in a 
given set of PMs of a datacenter m. The VMP problem is 
mostly formulated as a combinatorial optimization problem, 
representing one of the most challenging problems in virtu¬ 
alized datacenters infrastructure management, considering the 
large number of possible optimization criteria and different 
formulations that could be studied oni. 


For virtualized datacenters with deployments of VMs that 
rarely change its conhguration over time, a static (offline) 
formulation of the VMP problem may be appropriate CD. 
Additionally, in virtualized datacenters where a small number 
of VMs are created and destroyed, a semi-static formulation 
of the VMP problem could be acceptable (e.g. consolidating 
VMs every day at midnight). On the other hand, considering 
the today more realistic on-demand model of cloud computing 
with dynamic resource provisioning, a static (or semi-static) 
formulation of the VMP problem can result in under-optimal 
solutions after a short period of time. Clearly, the VMP prob¬ 
lem for cloud computing environments must be formulated as 
a pure dynamic (online) optimization problem to efficiently 
attend dynamic workload of modern applications Cl. 

A. Background and Motivation 

The VMP problem has been extensively studied and several 
surveys have already been presented in the VMP literature. 
Existing surveys focus on specihc issues such as: (1) energy- 
efficient techniques applied to the problem lO, ifTSl . (2) partic¬ 
ular architectures where the VMP problem is applied, as feder¬ 
ated clouds 0, and (3) methods for comparing performance of 
placement algorithms in large on-demand clouds CD. None 
of the mentioned surveys presented a general and extensive 
study of a large part of the VMP literature. In consequence, 
Lopez-Pires and Baran presented in ifTOl an extensive up-to- 
date survey of the most relevant VMP literature and proposed 
a novel taxonomy in order to identify research opportunities 
dehning a general vision on this problem. 

According to cni, the VMP problem is mostly formulated 
as an online optimization problem, where live migration 
techniques allow VMs to be dynamically consolidated on nec¬ 
essary PMs according to dynamic requirements of resources. 
The most studied environment for online formulations of the 
VMP problem considers that VMs are dynamically created and 
destroyed Go). To the best of the authors’ knowledge, there 
is no published work presenting a detailed characterization of 
possible dynamic environments for the VMP problem. 

Clearly, a deeper research of possible dynamic parameters 
in cloud computing is necessary in order to propose holistic 
and more realistic environments for the formulation of the 
VMP problem for cloud computing datacenters. 


Consequently, it is important to extend the taxonomy 
presented in cni focusing on dynamic formulations of the 
VMP problem from the providers’ perspective, in order to 
understand possible challenges for Cloud Service Providers 
(CSPs) in dynamic environments to efficiently attend cus¬ 
tomers’ requests for virtual resources, based on the most rele¬ 
vant dynamic parameters studied so far in the VMP literature. 

II. Reviewed Literature 

A. Keywords Search 

The selection process of relevant articles started with a 
search for research articles from Google Scholar database 
[scholar.google.com] with at least one of the following selected 
keywords in the article title: (1) virtual machine placement, 
(2) vm placement, (3) virtual machine consolidation, (4) vm 
consolidation or (5) server consolidation. The search of those 
keywords resulted in 446 research articles. A detailed list of 
the 446 resulting articles can be found in 13. 

B. Publisher Filtering 

Considering the large number of results from keywords 
search step, the literature selection process focused on research 
articles from the following well-known publishers: (1) ACM, 
(2) IEEE, (3) Elsevier and (4) Springer. This filtering step 
resulted in a reduction from 446 to 172 research articles. A 
detailed list of the 172 resulting articles can be found in E). 

C. Abstract Reading 

Considering the 172 resulting articles from the publisher 
filtering step, a reading of the abstracts was performed in order 
to identify the most relevant articles that specifically study 
the VMP problem. Additionally, short papers (i.e. research 
articles with less than 6 pages) were removed from the 
selected literature, resulting in 84 selected articles of the VMP 
literature. A detailed list of the 84 resulting articles can be 
found in Ig). 

D. Online Formulations for Provider-oriented VMP Problem 

Based on the 84 studied articles from Qol, this work 
focuses on the 64 articles that proposed online formulations for 
the VMP problem from the providers’ perspective, considering 
the relevance of this type of environments for actual cloud 
computing providers. An in-depth reading of this universe of 
64 articles was performed with the aim of identifying the most 
relevant dynamic parameters. 

III. Classieication Criteria 

This work identified the following dynamic parameters: 

• resource capacities of VMs (vertical elasticity); 

• number of VMs of a service (horizontal elasticity); 

• utilization of resources of VMs (related to overbooking). 

Consequently, dynamic environments for online formula¬ 
tions of the provider-oriented VMP problem may be classi¬ 
fied by one or more of the following classification criteria: 
(1) elasticity and (2) overbooking, as presented in the follow¬ 
ing subsections. 


A. Elasticity 

According to the definition given in ID: “Cloud elasticity 
is the ability of a cloud infrastructure to rapidly change the 
amount of resources allocated to a service in order to meet 
the actual varying demands on the service while enforcing a 
Service Level Agreement (SLA)”. 

Considering the dynamic workload of modern cloud appli¬ 
cations, proactive elasticity is a very important issue to address 
for CSPs in order to deal with under-provisioning (saturation) 
and over-provisioning (under-utilization) of cloud resources 
m- Under-provisioning can cause SLA violations, impacting 
directly on economical revenue while over-provisioning can 
cause inefficient utilization of resources, directly impacting 
on resource utilization and energy consumption. 

Research articles considering online formulations of the 
provider-oriented VMP literature have already studied two 
types of elasticity: vertical and horizontal (see Eigure [2. 
Vertical elasticity can be defined as the ability of cloud 
services to dynamically change capacities of virtual resources 
(e.g. CPU and RAM memory) inside a VM, while horizontal 
elasticity can be defined as the ability of cloud services to 
dynamically adjust the number of VMs associated to a cloud 
service Ea. 

Implementing vertical elasticity requires shorter time of 
service reconfiguration than horizontal elasticity, but with a 
higher migration cost. On the other hand, horizontal elasticity 
enables stronger high availability than vertical elasticity, but a 
coordination overhead is required and infrastructure complex¬ 
ity increases Ga. 

B. Overbooking 

Resources overbooking can make cloud services more 
profitable for CSPs, overlaying requested virtual resources 
onto physical resources at a higher ratio than 1:1 Q- Online 
formulations of the provider-oriented VMP considering over¬ 
booking include particular considerations to efficiently attend 
customers’ requirements, enforcing SLAs. 

Considering the dynamic workload of cloud applications 
and services, virtual resources of VMs are also dynamically 
used giving space to re-utilization of idle resources that 



Figure 1. Vertical and horizontal elasticity. Vertical elasticity dynamically ad¬ 
justs the capacities of virtual resources inside a VM while horizontal elasticity 
dynamically adjusts the number of VMs (e.g. in distributed applications). 















were already reserved. Research articles considering online 
formulations of the provider-oriented VMP literature have 
already studied two types of overbooking: server and network 
resources overbooking. 

In this context, CSPs should measure the utilization of 
resources of VMs in order to correctly manage the over¬ 
booking with the available physical resources, minimizing 
SLA violations. Monitoring utilization of virtual resources 
and workload of cloud applications and services also helps 
CSPs to consider forecasting techniques for approximating in 
advance the required management actions (e.g. migrations of 
VMs) for the consolidation process, to reduce resource under¬ 
provisioning 0 , ca. 


IV. Possible Classifications 


Based on the universe of 64 studied research articles, 
dynamic environments for online formulations of the provider- 
oriented VMP problem may be classified by one or more 
of the following classification criteria: (1) elasticity and (2) 


overbooking, as presented in Section III 


First, dynamic environments could be formulated consider¬ 
ing one of the following elasticity values: 

• elasticity=0: no elasticity; 

• elasticity=l: horizontal elasticity; 

• elasticity=2: vertical elasticity; 

• elasticity=3: horizontal and vertical elasticity. 
Additionally, identified dynamic environments may also 

consider one of the following overbooking values: 

• overbooking=0: no overbooking; 

• overbooking=l: server resources overbooking; 

• overbooking=2: network resources overbooking; 

• overbooking=3: server and network overbooking. 

Based on the combinations of the possible values of the 

classification criteria (elasticity,overbooking), it has been iden¬ 
tified 16 different possible environments. Considering this 
representation, each identified dynamic environment is denoted 
by its elasticity and overbooking coordinates. 


A. Cloud Service and Environment Notation 

CSPs dynamically receive requests for the placement of 
cloud services with different characteristics according to the 
classification criteria presented in Section m representing 
real-world environments and generalizing the deployment of 
cloud services in several possible cloud architectures (e.g. 
single-cloud, distributed-cloud or federated-cloud). Cloud ser¬ 
vices may represent simple services such as Domain Name 
Service (DNS) or complex multi-tier elastic applications. 

A cloud service is composed by a set of VMs, where each 
VM of a cloud service could be located for its execution in 
different cloud datacenters according to the customers prefer¬ 
ences or requirements (e.g. legal issues or high-availability). 

Configuration of VMs of a cloud service changes dy¬ 
namically when elasticity is considered. On the other hand, 
utilization of virtual resources change dynamically according 
to the demand when overbooking is considered; otherwise, the 
utilization of each virtual resource is considered at 100%. 


Formally, a cloud service Sb can be distributed across 
different possible cloud datacenters. Each cloud datacenter 
DCc hosts VMs V' associated to different cloud services. A 
VM pA associated to a service Si, is denoted as V^C. 


where: 


Sb'. 

Cloud service 6; 

DCc'. 

Cloud datacenter c; 

mDCc- 

Number of VMs Vj in DCc 

mSb- 

Number of VMs 14 - in Sb', 

K- 

Vj in DCc, 

Vi'cy 

Vj in DCc from service Sb- 


Figure 1^ presents a basic example of a cloud service ^i, 
distributed across 2 cloud datacenters DCi and DC 2 and 
using 4 VMs: V/Ji, Vli 2 ^ ^ 121 ’ ^ 122 - These cloud datacenters 
could represent geo-distributed datacenters owned by one CSP 
or a federated-cloud with two different CSPs. Each cloud 
datacenter hosts 2 VMs of Si'. and VI 2 represent Vi and 
V 2 in DCi respectively (analogously DC 2 hosts 2 VMs). 

Complementing the above notation, each cloud datacenter 
DCc may be represented as: 


DCc = {V:,,X2.---XmDC^ 


yi'c] ={ycpubcj , Vram'b'cj, VneX, Kcj . 

^^-^bcj ’ ^init ; lend} 


where: 


( 1 ) 

( 2 ) 


k;- 

Vcpu'Lf 


Vram 


Vnet[ 

TDff . 

^bcj' 

SLA 


hcj ■ 
bcj- 


bcj- 


s: 


linit • 
lend- 


Vj in DCc from service Sb', 

Processing requirements of in [ECU]; 
Memory requirements of 14 % in [GB]; 
Network requirements of in [Mbps]; 
Economical revenue for locating in [$]; 
SLA of 14%. SLAl^ G 4; 

Highest priority level of SLAs; 

Initial discrete time when 14 % is executed; 
Einal discrete time when 14 % is executed. 



Figure 2. Example of a cloud service considered in this work. 




















Table I 

Summary of time variables for 16 dynamic environments. 


ID 

Elasticity Type 

Overbooking Type 

(0.0) 

Not Considered 

Not Considered 

(0,1) 

Not Considered 

Server 

(0.2) 

Not Considered 

Network 

(0,3) 

Not Considered 

Server and Network 

(1.0) 

Horizontal 

Not Considered 

(1,1) 

Horizontal 

Server 

(1,2) 

Horizontal 

Network 

(1,3) 

Horizontal 

Server and Network 

(2.0) 

Vertical 

Not Considered 

(2,1) 

Vertical 

Server 

(2,2) 

Vertical 

Network 

(2,3) 

Vertical 

Server and Network 

(3,0) 

Horizontal and Vertical 

Not Considered 

(3,1) 

Horizontal and Vertical 

Server 

(3.2) 

Horizontal and Vertical 

Network 

(3,3) 

Horizontal and Vertical 

Server and Network 


Utilization of the resources of each is represented by: 
U'bcj = {Ucpu'l^j, Uram'l^^,Unet'l^j} (3) 

where: 

t/jU: Utilization of requirement 14"^ ; 

Ucpu'l^j-. Utilization of Vcpu'^^j in [ECU]; 

Uram'^^j: Utilization of Vram'^^j in [GB]; 

Unet'l^y Utilization of Vnet'^^j in [Mbps]. 

Note that in practical applications C/^U is lower than 
giving place to overbooking of resources. 

Each of the 16 identified environments considers different 
parameters that dynamically change as a function of time t, 
giving place to possible different notations for each environ¬ 
ment as presented in Table 

B. Dynamic Environment Examples 

This section will show representative examples with a 
detailed explanation of 4 of the 16 different dynamic envi¬ 
ronments. 

Example 1 is presented in Eigure showing a detailed 
Environment (0,1). The first level is CSPi, it represents a 
CSP (Cloud Service Provider) requests for allocating VMs. 
Eollowing, shows the requested VMs for the cloud service. 
DCi and DC 2 present the resources that the cloud service 
needs along every discrete time. Because this environment do 
not considered elasticity, the number of VMs is maintained 
from time t = 0 to f = 5. However, overbooking of 
server resources is considered and this figure shows how the 
utilization of CPU and memory varies for every VM in the 
cloud service. The first VM (Um) samples an increase of 
Ucpuiii and Uramm for every discrete time, while U 121 
shows the decrease of Ucpui 2 i from f = 3 to f = 4 and an 
increase of Urami 2 i at the same period time. Other variations 
of this resources for every VM, executed in this environment, 
can be observed in this figure. 


Example 2 is presented following with the detailed expla¬ 
nation of the environments, Eigure exposes overbooking 
of network resources without considering vertical and hor¬ 
izontal elasticity. Analogously to the previews environment, 
no increase in the number of VMs is presented. A dynamic 
utilization of network resources is presented in V 121 with a 
sharp increase of Uneti 2 i from f = 0 to f = 5. Other VMs in 
this environment also present variations of network utilization. 

Example 3 Eigure shows a detailed example of require¬ 
ments for Environment (1,0). Knowing that this environment 
takes into account horizontal elasticity, the table shows the 
variation of the request of VMs along t = 0 and t = 5. 
Erom f = 0 to f = 1, CSPi executes only four VMs. At the 
next time unit, the request that the CSP receive changes from 
two VMs to four VMs. The figure shows that at f = 2, both 
datacenters (DCi and DC 2 ) needed another resources, for 
this reason, S 2 was created with one VM per each datacenter 
(U 3 for DCi and V 3 for DC 2 )- This example shows the 
resources and requirements of each VM that are executed in 
this environment. The mentioned scenario do not considered 
overbooking and vertical elasticity, for this reason, the VM 
requirements and utilization resources stays equal from f = 0 
to f = 5. 

Example 4 The last example, Eigure shows a dynamic 
environment that considers vertical elasticity and no overbook¬ 
ing, Environment (2,0). Eor V 112 , the requirements Vcpuii 2 
and Vramii 2 increase in order to cover the needs of the cloud 
service. Every VM, detailed in this figure, shows the variation 
that the requirements present along t = 0 and t = 5. 

The 12 remaining environments are composed by the com¬ 
bination of this 4 previously explained environments. Those 
12 environments were not detailed with examples due to simi¬ 
larity and content with the previously explained environments. 

V. Conclusions and Euture Directions 
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Figure 3. Detailed example of dynamic Environment (1,0) for 5 instants of time (t) 
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Figure 4. Detailed example of dynamic Environment (0,2) for 5 instants of time (t) 
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Figure 5. Detailed example of dynamic Environment (1,0) for 5 instants of time (f) 
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Figure 6. Detailed example of dynamic Environment (2,0) for 5 instants of time (t) 























































